Antenna Season Report Notebook¶

Josh Dillon, Last Revised January 2022

This notebook examines an individual antenna's performance over a whole season. This notebook parses information from each nightly rtp_summarynotebook (as saved to .csvs) and builds a table describing antenna performance. It also reproduces per-antenna plots from each auto_metrics notebook pertinent to the specific antenna.

In [1]:
import os
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the next line of this cell.
# antenna = "004"
# csv_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/_rtp_summary_'
# auto_metrics_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/auto_metrics_inspect'
# os.environ["ANTENNA"] = antenna
# os.environ["CSV_FOLDER"] = csv_folder
# os.environ["AUTO_METRICS_FOLDER"] = auto_metrics_folder
In [3]:
# Use environment variables to figure out path to the csvs and auto_metrics
antenna = str(int(os.environ["ANTENNA"]))
csv_folder = os.environ["CSV_FOLDER"]
auto_metrics_folder = os.environ["AUTO_METRICS_FOLDER"]
print(f'antenna = "{antenna}"')
print(f'csv_folder = "{csv_folder}"')
print(f'auto_metrics_folder = "{auto_metrics_folder}"')
antenna = "118"
csv_folder = "/home/obs/src/H6C_Notebooks/_rtp_summary_"
auto_metrics_folder = "/home/obs/src/H6C_Notebooks/auto_metrics_inspect"
In [4]:
display(HTML(f'<h1 style=font-size:50px><u>Antenna {antenna} Report</u><p></p></h1>'))

Antenna 118 Report

In [5]:
import numpy as np
import pandas as pd
pd.set_option('display.max_rows', 1000)
import glob
import re
from hera_notebook_templates.utils import status_colors, Antenna
In [6]:
# load csvs and auto_metrics htmls in reverse chronological order
csvs = sorted(glob.glob(os.path.join(csv_folder, 'rtp_summary_table*.csv')))[::-1]
print(f'Found {len(csvs)} csvs in {csv_folder}')
auto_metric_htmls = sorted(glob.glob(auto_metrics_folder + '/auto_metrics_inspect_*.html'))[::-1]
print(f'Found {len(auto_metric_htmls)} auto_metrics notebooks in {auto_metrics_folder}')
Found 14 csvs in /home/obs/src/H6C_Notebooks/_rtp_summary_
Found 13 auto_metrics notebooks in /home/obs/src/H6C_Notebooks/auto_metrics_inspect
In [7]:
# Per-season options
mean_round_modz_cut = 4
dead_cut = 0.4
crossed_cut = 0.0

def jd_to_summary_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/_rtp_summary_/rtp_summary_{jd}.html'

def jd_to_auto_metrics_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/auto_metrics_inspect/auto_metrics_inspect_{jd}.html'

Load relevant info from summary CSVs¶

In [8]:
this_antenna = None
jds = []

# parse information about antennas and nodes
for csv in csvs:
    df = pd.read_csv(csv)
    for n in range(len(df)):
        # Add this day to the antenna
        row = df.loc[n]
        if isinstance(row['Ant'], str) and '<a href' in row['Ant']:
            antnum = int(row['Ant'].split('</a>')[0].split('>')[-1]) # it's a link, extract antnum
        else:
            antnum = int(row['Ant'])
        if antnum != int(antenna):
            continue
        
        if np.issubdtype(type(row['Node']), np.integer):
            row['Node'] = str(row['Node'])
        if type(row['Node']) == str and row['Node'].isnumeric():
            row['Node'] = 'N' + ('0' if len(row['Node']) == 1 else '') + row['Node']
            
        if this_antenna is None:
            this_antenna = Antenna(row['Ant'], row['Node'])
        jd = [int(s) for s in re.split('_|\.', csv) if s.isdigit()][-1]
        jds.append(jd)
        this_antenna.add_day(jd, row)
        break
In [9]:
# build dataframe
to_show = {'JDs': [f'<a href="{jd_to_summary_url(jd)}" target="_blank">{jd}</a>' for jd in jds]}
to_show['A Priori Status'] = [this_antenna.statuses[jd] for jd in jds]

df = pd.DataFrame(to_show)

# create bar chart columns for flagging percentages:
bar_cols = {}
bar_cols['Auto Metrics Flags'] = [this_antenna.auto_flags[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jee)'] = [this_antenna.dead_flags_Jee[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jnn)'] = [this_antenna.dead_flags_Jnn[jd] for jd in jds]
bar_cols['Crossed Fraction in Ant Metrics'] = [this_antenna.crossed_flags[jd] for jd in jds]
bar_cols['Flag Fraction Before Redcal'] = [this_antenna.flags_before_redcal[jd] for jd in jds]
bar_cols['Flagged By Redcal chi^2 Fraction'] = [this_antenna.redcal_flags[jd] for jd in jds]
for col in bar_cols:
    df[col] = bar_cols[col]

z_score_cols = {}
z_score_cols['ee Shape Modified Z-Score'] = [this_antenna.ee_shape_zs[jd] for jd in jds]
z_score_cols['nn Shape Modified Z-Score'] = [this_antenna.nn_shape_zs[jd] for jd in jds]
z_score_cols['ee Power Modified Z-Score'] = [this_antenna.ee_power_zs[jd] for jd in jds]
z_score_cols['nn Power Modified Z-Score'] = [this_antenna.nn_power_zs[jd] for jd in jds]
z_score_cols['ee Temporal Variability Modified Z-Score'] = [this_antenna.ee_temp_var_zs[jd] for jd in jds]
z_score_cols['nn Temporal Variability Modified Z-Score'] = [this_antenna.nn_temp_var_zs[jd] for jd in jds]
z_score_cols['ee Temporal Discontinuties Modified Z-Score'] = [this_antenna.ee_temp_discon_zs[jd] for jd in jds]
z_score_cols['nn Temporal Discontinuties Modified Z-Score'] = [this_antenna.nn_temp_discon_zs[jd] for jd in jds]
for col in z_score_cols:
    df[col] = z_score_cols[col]

ant_metrics_cols = {}
ant_metrics_cols['Average Dead Ant Metric (Jee)'] = [this_antenna.Jee_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Dead Ant Metric (Jnn)'] = [this_antenna.Jnn_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Crossed Ant Metric'] = [this_antenna.crossed_metrics[jd] for jd in jds]
for col in ant_metrics_cols:
    df[col] = ant_metrics_cols[col]

redcal_cols = {}
redcal_cols['Median chi^2 Per Antenna (Jee)'] = [this_antenna.Jee_chisqs[jd] for jd in jds]
redcal_cols['Median chi^2 Per Antenna (Jnn)'] = [this_antenna.Jnn_chisqs[jd] for jd in jds]   
for col in redcal_cols:
    df[col] = redcal_cols[col]

# style dataframe
table = df.style.hide_index()\
          .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
          .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=list(z_score_cols.keys())) \
          .background_gradient(cmap='bwr_r', vmin=dead_cut-.25, vmax=dead_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .background_gradient(cmap='bwr_r', vmin=crossed_cut-.25, vmax=crossed_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .background_gradient(cmap='plasma', vmax=4, vmin=1, axis=None, subset=list(redcal_cols.keys())) \
          .applymap(lambda val: 'font-weight: bold' if val < dead_cut else '', subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val < crossed_cut else '', subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .applymap(lambda val: 'color: red' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .bar(subset=list(bar_cols.keys()), vmin=0, vmax=1) \
          .format({col: '{:,.4f}'.format for col in z_score_cols}) \
          .format({col: '{:,.4f}'.format for col in ant_metrics_cols}) \
          .format('{:,.2%}', na_rep='-', subset=list(bar_cols.keys())) \
          .set_table_styles([dict(selector="th",props=[('max-width', f'70pt')])]) 

Table 1: Per-Night RTP Summary Info For This Atenna¶

This table reproduces each night's row for this antenna from the RTP Summary notebooks. For more info on the columns, see those notebooks, linked in the JD column.

In [10]:
display(HTML(f'<h2>Antenna {antenna}, Node {this_antenna.node}:</h2>'))
HTML(table.render(render_links=True, escape=False))

Antenna 118, Node N07:

Out[10]:
JDs A Priori Status Auto Metrics Flags Dead Fraction in Ant Metrics (Jee) Dead Fraction in Ant Metrics (Jnn) Crossed Fraction in Ant Metrics Flag Fraction Before Redcal Flagged By Redcal chi^2 Fraction ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score Average Dead Ant Metric (Jee) Average Dead Ant Metric (Jnn) Average Crossed Ant Metric Median chi^2 Per Antenna (Jee) Median chi^2 Per Antenna (Jnn)
2459827 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 2.440443 29.505855 2.769368 31.025237 -0.031438 26.692914 -0.193162 -0.948185 0.7554 0.0484 0.4107 8.400148 1.270718
2459826 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 2.186042 29.029873 2.950854 35.174598 -0.926287 45.570619 -0.215537 3.069430 0.8065 0.0491 0.4893 0.000000 0.000000
2459825 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 1.753405 31.518340 1.634076 28.282420 -1.098361 25.789033 -0.722465 -0.310416 0.8141 0.0493 0.4015 4.835412 1.268158
2459824 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 1.184588 22.073296 2.158959 21.811973 -0.552542 19.548152 0.177300 -0.025019 0.7128 0.0481 0.3180 5.430461 1.321672
2459823 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 2.541613 27.536644 2.933803 42.490663 1.606483 36.032293 -0.468575 25.210659 0.7738 0.0507 0.3549 54.166641 1.333457
2459822 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 2.686601 29.773009 2.296176 38.562312 -0.447120 29.723044 0.106859 -0.103968 0.8033 0.0503 0.4122 4.809486 1.273519
2459821 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 3.021764 33.366638 2.266414 39.192383 -0.729142 26.038175 -1.170894 -1.227518 0.8092 0.0442 0.4662 4.885820 1.238810
2459820 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 4.185083 29.146308 2.447325 31.061740 -0.474223 70.714619 2.503867 0.635577 0.7631 0.0517 0.4791 4.403567 1.300430
2459817 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 2.771247 28.679154 1.557459 38.392574 0.085004 36.772078 0.116977 0.704878 0.8112 0.0532 0.5227 4.057866 1.321839
2459816 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 3.253228 23.599998 2.741401 38.498190 1.560346 45.704915 1.076637 2.525416 0.8541 0.0520 0.6463 5.231166 1.270027
2459815 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 3.101329 25.153446 2.568122 42.247765 1.947453 48.235579 -0.514039 7.100405 0.8174 0.0499 0.5914 5.196134 1.238075
2459814 digital_ok 0.00% - - - - - nan nan nan nan nan nan nan nan nan nan nan nan nan
2459813 digital_ok 100.00% 0.00% 100.00% 0.00% 100.00% 0.00% 5.368763 30.596226 1.808102 25.785123 1.015559 87.067970 3.131879 2.187284 0.7815 0.0440 0.3928 12.009362 1.363353

Load antenna metric spectra and waterfalls from auto_metrics notebooks.¶

In [11]:
htmls_to_display = []
for am_html in auto_metric_htmls:
    html_to_display = ''
    # read html into a list of lines
    with open(am_html) as f:
        lines = f.readlines()
    
    # find section with this antenna's metric plots and add to html_to_display
    jd = [int(s) for s in re.split('_|\.', am_html) if s.isdigit()][-1]
    try:
        section_start_line = lines.index(f'<h2>Antenna {antenna}: {jd}</h2>\n')
    except ValueError:
        continue
    html_to_display += lines[section_start_line].replace(str(jd), f'<a href="{jd_to_auto_metrics_url(jd)}" target="_blank">{jd}</a>')
    for line in lines[section_start_line + 1:]:
        html_to_display += line
        if '<hr' in line:
            htmls_to_display.append(html_to_display)
            break

Figure 1: Antenna autocorrelation metric spectra and waterfalls.¶

These figures are reproduced from auto_metrics notebooks. For more info on the specific plots and metrics, see those notebooks (linked at the JD). The most recent 100 days (at most) are shown.

In [12]:
for i, html_to_display in enumerate(htmls_to_display):
    if i == 100:
        break
    display(HTML(html_to_display))

Antenna 118: 2459827

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Power 31.025237 2.440443 29.505855 2.769368 31.025237 -0.031438 26.692914 -0.193162 -0.948185

Antenna 118: 2459826

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Temporal Variability 45.570619 29.029873 2.186042 35.174598 2.950854 45.570619 -0.926287 3.069430 -0.215537

Antenna 118: 2459825

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Shape 31.518340 31.518340 1.753405 28.282420 1.634076 25.789033 -1.098361 -0.310416 -0.722465

Antenna 118: 2459824

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Shape 22.073296 1.184588 22.073296 2.158959 21.811973 -0.552542 19.548152 0.177300 -0.025019

Antenna 118: 2459823

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Power 42.490663 27.536644 2.541613 42.490663 2.933803 36.032293 1.606483 25.210659 -0.468575

Antenna 118: 2459822

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Power 38.562312 2.686601 29.773009 2.296176 38.562312 -0.447120 29.723044 0.106859 -0.103968

Antenna 118: 2459821

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Power 39.192383 33.366638 3.021764 39.192383 2.266414 26.038175 -0.729142 -1.227518 -1.170894

Antenna 118: 2459820

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Temporal Variability 70.714619 4.185083 29.146308 2.447325 31.061740 -0.474223 70.714619 2.503867 0.635577

Antenna 118: 2459817

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Power 38.392574 2.771247 28.679154 1.557459 38.392574 0.085004 36.772078 0.116977 0.704878

Antenna 118: 2459816

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Temporal Variability 45.704915 23.599998 3.253228 38.498190 2.741401 45.704915 1.560346 2.525416 1.076637

Antenna 118: 2459815

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Temporal Variability 48.235579 25.153446 3.101329 42.247765 2.568122 48.235579 1.947453 7.100405 -0.514039

Antenna 118: 2459814

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Shape nan nan nan nan nan nan nan nan nan

Antenna 118: 2459813

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
118 N07 digital_ok nn Temporal Variability 87.067970 30.596226 5.368763 25.785123 1.808102 87.067970 1.015559 2.187284 3.131879

In [ ]: